This invention relates generally to test systems for disk storage system peripherals, and more particularly for testing systems for storage systems that use RAID technology.
Users who purchase RAID (Redundant Array of Independent Disks) storage systems, usually do so because they have large amounts of important data to store in a quickly accessible, affordable, reliable online form. RAID systems were developed for this purpose. In most RAID systems, several disks are used together in an array, and treated as logical disks by array management software or circuitry. Reliability and performance are handled by array management software in a disk controller or on the host computer.
Testing the array management software and the RAID system's ability to detect and handle various types of disk and media failures is made much more complicated by the complexity of the various types of RAID levels.
Different types of RAID systems, (RAID levels 0, 1, 2, 3, 4, and 5) offer users different levels and combinations of data security and access speed or performance. RAID 0, for example, uses a striping system to stripe data across several physical disks, with one part or "chunk" of the data on each disk. Level 1 mirrors the data, instead of striping it, by storing at least one extra copy of it on another disk in the array, to provide very high data reliability. RAID levels 2 and 3 provide high data transfer capabilities for files that are large, and read in large, continuous streams. They both assume parallel access of the files. RAID level 2 uses a Hamming code to detect and correct disk errors. In Raid level 3, parity data is stored on one disk. If that disk fails, the data is still intact, but the security provided by parity checking is lost. If a data disk fails, the data can be regenerated using the parity disk and the other disks in the system.
RAID level 4 is similar to level 3, in that it has data mapped across disks, in chunks, with one disk reserved for parity data, but level 4 operates its disks independently, while, in level 3, disks are operated in parallel. RAID level 5 is similar to 4, but instead of placing parity data on one disk, parity chunks are striped across the member disks. RAID-5 is, in some ways, the most complex type of RAID system.
A RAID-5 system is expected to be highly available, to handle any single block error or any single disk failure, and get good performance for numerous reads and writes of short to medium length data. If two failures occur, however, the system may not be able to recover the data, but a RAID level 5 implementation is still supposed to detect that error occurred and let the user know.
In RAID level 5, parity is the exclusive or (.sym.) function applied to the data chunks. Here, the term stripe is used to define the relationship of a parity chunk, and a collection of adjunct data chunks (each of which may be located on different drives), that are associated with each other as the content of their data is used in the calculation of parity data and the possible reconstruction of any associated chunk experiencing a media error or device failure. If data is striped in 4 data chunks across several drives, the parity chunk for that stripe is computed as: EQU P(0-3)=Chunk 0.sym.Chunk 1.sym.Chunk 2.sym.Chunk 3
Any one chunk of the data can be regenerated from the contents of the corresponding chunks in that stripe on the remaining disks in the array. Thus, the operation EQU x=P(0-3).sym.Chunk 0.sym.Chunk 1.sym.Chunk 2
removes the effects of Chunks 0, 1, and 2 from the parity comparison and leaves x=Chunk 3. This is how RAID 5 systems are able to recover from a single disk or media failure. If one disk is down, or a block is defective, data on that disk can be reconstructed by the RAID 5 algorithm, by exclusive or'ing the remaining chunks of that stripe. If two failures occur together in a stripe, say disk one is completely down, and several blocks on disk 3 of a 5 disk system are bad, a RAID-5 system will not be able to reconstruct the requested blocks, but it should be able to detect this situation and report it to the user.
In order to test the array management function to insure that it correctly reconstructs data or detects unrecoverable double failures, a variety of correctable and non-correctable errors of various types and combinations need to be presented to the array management software.
Before the advent of RAID systems, one could present hard disk errors to systems and system software by manually powering a given disk drive down, or by inserting a disk that had known defects at known tracks. If the disk drive controller firmware or system software detected these situations and handled them properly, the test was a success. However, the very sophistication which makes RAID array management able to recover from or detect many more types of errors, makes it much harder to test the software or firmware itself.
For example, a RAID-5 system is supposed to recover from all single failures and at least detect and report any double failures. Many double failure scenarios can also be corrected. If a tester tries to create a double failure by placing two damaged disks into the array, it is difficult to organize a stripe so that the failures will occur together within the same stripe. Similarly, if a tester attempted to introduce a failure by powering down one drive, at the time it is estimated another drive with a known defective block will be read for that stripe, this is not likely to be an effective test, since the operator works in seconds, while the drives and systems are operating in milliseconds.
Using defective disks with known bad blocks doesn't provide much of a test even for the single failure algorithms. In addition, the test may not be consistently repeatable, given the difficulty of insuring that the array management software will always place the chunks of a stripe in such a way that they will consistently fall in the area known to be damaged.
Manufacturing testers have automated tests of error-handling for single disk non-RAID systems, by simulating or emulating error conditions that are likely to occur. RAID algorithms, however, are far more complicated than those for single disk systems, and attempts to automate this part of testing have met with frustration. A further challenge is presented when the RAID 5 systems uses cache to improve performance. For example, a data block that was corrupted on the disk, may be overwritten by pending data in the cache, so that when a read command is executed it retrieves the overwritten, good data not the once corrupted block. In RAID-5 systems, a cache may also cause problems for tests of write commands, since another aspect of a RAID-5 system is that a write actually causes a chunk and its corresponding parity chunk to be read first, so that the old data and parity can be exclusive or'd with the new data for parity and then the physical writes to the disk occur. If the test system cannot control or guarantee the cache contents it is difficult to know if the desired error was, in fact, physically created on the disk.
In the same vein, a set of programs that automatically issue command sequences that might adequately test a RAID 0 system, could be nearly worthless for a RAID 4 or RAID 5 system, since the types of error patterns and availability goals each level is designed to handle are so different.
For example, to test a RAID-5 system's ability to handle defects on the media, by corrupting a chunk of a stripe, you need to know where the array management software will place the stripe patterns on the drives, so you will know which chunk of which stripe to corrupt. Trying to do this manually would be prohibitively time-consuming, and even writing a program to do this for RAID-5 would not be useful for RAID-1. In a RAID-1 system data is not striped.
A parity test for RAID-5 would be to present it with a corrupt parity block from the disk and see if it recovers the data. However, test systems that run on the host computer that the RAID system is attached to, normally can't corrupt the parity block, since parity is not visible to the host.
Since a primary function of all RAID systems is to provide some level of redundancy or reliability for the data, RAID array management that is defective can be costly for a manufacturer and its customers. Single disk systems can only store a few gigabytes of data. Very large RAID systems may cost hundreds of thousands of dollars and store hundreds of gigabytes of data. Other RAID systems may be used in client/server networks and a failure may mean costly outages for all personnel and work on the network affected by it.
Testing that can detect defects in RAID array management and permit them to be corrected before the product is shipped can be both useful and valuable. If such testing could be reproduced reliably, it could help in regression testing of the same array management function when new versions or upgrades are developed.
It is an object of the present invention to provide a method and apparatus for testing RAID array management.
It is another object of the present invention to provide a method and apparatus for automating the testing of RAID array management software.
Still another object of the present invention is providing a method and apparatus for testing RAID array management that can be done in a repeatable manner so that regression tests can be performed.
Yet another object of the present invention is providing a method and apparatus for testing RAID array management that can be readily adapted to include tests for new features or new combinations of tests or test situations.